Determining and utilizing secondary language proficiency measure

ABSTRACT

Implementations relate to determining a secondary language proficiency measure, for a user in a secondary language (i.e., a language other than a primary language specified for the user), where determining the secondary language proficiency measure is based on past interactions of the user that are related to the secondary language. Those implementations further relate to utilizing the determined secondary language proficiency measure to increase efficiency of user interaction(s), such as interaction(s) with a language learning application and/or an automated assistant. Some of those implementations utilize the secondary language proficiency measure in automatically setting value(s), biasing automatic speech recognition, and/or determining how to render natural language output.

BACKGROUND

A user may have interest in having content that is not in the samelanguage as the primary language of the user provided in response tointeractions of the user. For example, a user may utter one or morephrases in a secondary language, to an automated assistant, that wouldrequire recognition of the utterance in a language other than theprimary language of the user. In many instances, an automatic speechrecognition model is configured to recognize utterances in the primarylanguage of the user and is less likely to recognize utterances in asecondary language. In those instances, the automated assistant may beunable to respond to such utterances and/or may misinterpret theutterances by attempting to understand the utterances as terms in theprimary language of the user.

As another example, a user may have interest in learning a secondarylanguage but may not effectively identify resources for learning thelanguage that are directed to users with the proficiency of the userand/or other knowledge of the user. For example, a user may be providedwith translations of a phrase but the phrase may be too complex for auser to fully comprehend in its entirety. This can result in the usernot utilizing resources of an automated assistant that would otherwisebe capable of assisting in the language learning process.

SUMMARY

Implementations disclosed herein relate to determining a secondarylanguage proficiency measure, for a user in a secondary language (e.g.,a language other than a primary language specified for the user and/or alanguage that the user has interest in learning), where determining thesecondary language proficiency measure is based on past interactions ofthe user that are related to the secondary language. Thoseimplementations further relate to utilizing the determined secondarylanguage proficiency measure to increase efficiency of one or more userinteractions (e.g., interaction(s) with a language learning applicationand/or an automated assistant).

In some instances, a user may have multiple primary languages. Forexample, a user may be fluent in both English and French, and maytherefore be capable of submitting well-formed phrases in bothlanguages. Further, in some instances, a user may have one or moresecondary languages in which the user is less fluent. For example, auser may be currently learning both Spanish and Italian, and bothlanguages may be indicated as secondary languages for the user. Further,a user may specify a language as a primary language in instances wherethe user intends to converse in a language but the language is still asecondary language for the user. For example, a user may be a speaker ofEnglish but indicate that French is a primary language. One or moresignals can be utilized to determine that the user is learning French(i.e., the language is the second language of the user) even though theuser has indicated English as a primary language.

As one example, the secondary language proficiency measure can beutilized to automatically set value(s) for one or more parameters ofadjustable settings of a graphical user interface (GUI) of a languagelearning application. For instance, a value of “Spanish” can beautomatically set for a user that has interest, based on the pastinteractions of the user, in learning Spanish and/or a value of“beginner” can be automatically set for a language proficiency level forthe user. Automatically setting the value(s), in view of the secondarylanguage proficiency measure, obviates user input(s) that wouldotherwise be required to manually set the value(s) and enables userinteraction with the GUI of the language learning application toconclude more quickly. Moreover, in various implementations thesecondary language proficiency measure can be generated based on (e.g.,based solely on) interactions that are not interactions with thelanguage learning application. This enables automatically setting thevalue(s) in an initial interaction of the user with the GUI and/orenables the value(s) to be automatically dynamically adjusted, overtime, in view of interactions that are external to the language learningapplication. In addition to the computational efficiencies achievedthrough obviating input(s) with the GUI and/or enabling quickerinteraction with the GUI, automatically setting the value(s) mitigatesoccurrences of the language learning application utilizing, ininteractions with the user, inaccurate settings. Utilization ofinaccurate settings by the language learning application can result incomputationally wasteful interactions with the user such as rendering ofsecondary language output that is too simplistic or too complex to be ofbenefit to the user.

As another example, the secondary language proficiency measure canadditionally or alternatively be utilized to increase the accuracy ofautomatic speech recognition (ASR) that is performed, using an ASR modelfor the secondary language, on secondary language spoken utterances ofthe user that are in the secondary language. For instance,implementations select, from a superset of terms for the secondarylanguage, a subset of terms for the secondary language based on thedetermined secondary language proficiency measure, and utilize theselected subset of terms to bias ASR that is performed for secondarylanguage utterances of the user. The subset of terms can include, forexample, terms that the user is likely to be learning based on thesecondary language proficiency measure and can optionally excludeterm(s) that the user is unlikely to be learning and/or has alreadymastered. In these and other manners, spoken utterances of the user aremore likely to be accurately recognized, preventing occurrences oferrant interactions that are based on inaccurate ASR and/or preventingoccurrences of a repeat of spoken utterances that were inaccuratelyrecognized. The secondary language ASR can be utilized in processingspoken utterances of the user that are provided in interactions with thelanguage learning application, in general interactions of the user withan automated assistant, and/other interactions of the user.

As another example, the secondary language proficiency measure canadditionally or alternatively be utilized to determine how to render(e.g., audibly and/or visually), to the user, natural language outputthat is in the secondary language. For example, natural language outputtext can be split into a plurality of subtexts, where a quantity of thesubtexts and/or an amount of text in each of the subtexts can bedetermined based on the secondary language proficiency measure. Further,each of the subtexts can be rendered with a demarcation relative toother of the subtexts. For instance, with visual rendering each of thesubtexts can be rendered as a visually demarcated separate block, suchas a block that is vertically offset relative to the rendering of anypreceding and/or following block. Also, for instance, with audiblerendering through speech synthesis of the text, a demarcating pause inspeech and/or a demarcating earcon can be rendered between each of thesubtexts. Generally, more subtexts and/or smaller size subtexts will beprovided with a lower secondary language proficiency measure whilelesser subtexts and/or larger size subtexts will be provided with ahigher secondary language proficiency measure. In these and othermanners, secondary language output is rendered in view of the secondarylanguage proficiency measure. This can make the output moreunderstandable (through more and/or smaller subtexts) for a lesserproficiency user, while making the rendering shorter (in the case ofaudible rendering) and/or take up less screen real estate (in the caseof visual rendering) for a higher proficiency user. Accordingly, theoutput can be adapted to increase comprehension where appropriate, whilepromoting audible and/or visual efficiencies where appropriate. Suchadaptations with rendering can occur in natural language output that isgenerated by the language learning application and/or in interactionswith other application(s), such as general automated assistantinteractions with an automated assistant application (e.g., where theautomated assistant application may provide both a primary languageresponse and a secondary language response).

A user can select a primary language for an account to indicate thelanguage that the user prefers to primarily utilize for providingqueries and receiving responses to queries. In some instances, the usermay have interest in learning a secondary language (i.e., a languageother than the language selected as the primary language of the user)and can utilize one or more tools, such as a language learningapplication, to assist in learning the language. Further, the user mayprovide audio data that the user has interest in being translated to thesecondary language and/or may provide audio data that is in thesecondary language of the user. However, to initiate usage of languagelearning tools and/or to indicate the level of language proficiency ofthe user, a language learning application and/or other functionality mayrequire a setup and/or registration process, which may include settinginput parameters in order for the application to provide tools forlearning the secondary language at the proficiency level of the user.For example, in some instances, the user may have little or noproficiency in the secondary language, which may require simplevocabulary and grammar to be provided to the user in order for the userproficiency to improve. In other instances, the user may have someproficiency with the secondary language (e.g., some knowledge ofvocabulary and grammar rules), and therefore have interest in beingprovided with language learning tools that are more advanced than thoseprovided to a user that is a beginner in the secondary language.

By determining a secondary language proficiency measure for a user, alanguage learning application can be pre-populated with settings and/orparameters that indicate the secondary language proficiency of the user.Further still, the determined language proficiency measure canadditionally and/or alternatively be utilized to bias a speechrecognition model such that the speech recognition model can moreaccurately recognize terms spoken in the secondary language whenidentified in audio data provided by the user. Further still, thesecondary language proficiency measure can be utilized in determining toprovide translations of phrases in the secondary language such that theuser is provided with subtexts of a phrase as individual translationsinstead of the entire phrase as one single translated phrase.

In some implementations, a language proficiency measure for the user maybe determined based on past interactions of the user. The interactionscan be between the user and one or more applications. In some instances,the secondary language proficiency measure can be determined based onthe past interactions with particular applications. For example,interactions of the user with select applications may be utilized todetermine a secondary language proficiency measure, while interactionsof the user with other applications may not be utilized to determine thesecondary language proficiency measure. As an example, in someimplementations disclosed herein, past interactions of the user with oneor more applications other than a secondary language learningapplication can be utilized to determine a secondary languageproficiency measure.

In some implementations, one or more queries that are provided by theuser via a spoken utterance may be utilized to determine a secondarylanguage proficiency measure for the user. For example, the choice ofvocabulary of the user, when submitting spoken queries in the secondarylanguage, may be indicative of the proficiency of the user. Accordingly,the secondary language proficiency measure can be determined thatreflects the limited vocabulary of the user. Also, for example,grammatical and/or pronunciation errors of the user, when submittingspoken queries in the secondary language, may be indicative of thesecondary language proficiency of the user. Accordingly, a secondarylanguage proficiency measure may be determined for a user that submitqueries with grammatical and/or pronunciation errors that is lessindicative of proficiency than a user that submit spoken queries thatare grammatically correct.

In some implementations, interactions of the user that include the usersubmitting text in the secondary language may be utilized to determine asecondary language proficiency measure for the user. For example, withpermissions from the user, emails of the user that include terms and/orphrases in the secondary language may be identified and, based on thecorrectness of the grammar and/or the selection of terms in the emails(or other documents) can be utilized to determine a secondary languageproficiency measure for the user. Also, for example, auto-suggestionsthat are provided to the user when submitting queries can be utilized todetermine a secondary language proficiency measure (e.g., common termsthat are autocorrected can result in a determined secondary languageproficiency measure that is less indicative of proficiency than a userthat does not require autocorrect for the same common terms and/orphrases).

In some implementations, a secondary language proficiency measure can bedetermined for a user based on the user submitting queries (e.g., spokenqueries and/or text queries) that are related to the secondary languagebut provided in the primary language of the user. For example, a usermay submit a request to translate a phrase from the primary language ofthe user to the secondary language. The secondary language proficiencymeasure can be determined based on complexity of the phrases such that,for example, a user that submits requests for common terms may have alower determined proficiency measure than a user that submits requeststo translate more complex phrases.

In some implementations, user requests for additional informationrelated to the secondary language can be utilized to determine asecondary language proficiency measure for the user. In someimplementations, web activity of the user may be utilized to determine asecondary language proficiency measure for the user. For example, inaddition to submitting requests to translate terms into the secondarylanguage and/or submitting queries in the secondary language, the usermay submit a request to be provided with additional resources related tolearning the secondary language, such as “what is the best way to learnSpanish” or “take me to a website for learning French.” Also, forexample, the user may navigate to a webpage that is related to learningthe secondary language. Also, for example, the user may navigate towebpages that include content that is in the secondary language. In someimplementations, interactions of the user with one or more languagelearning applications can be utilized to determine a secondary languageproficiency measure for the user. For example, a user may download anapplication that is utilized to learn the secondary language, interactwith one or more applications that are related to learning the secondarylanguage, and/or otherwise utilize an application that, through theinteractions of the user, may be used to determine a secondary languageproficiency measure for the user.

Based on the secondary language proficiency measure, a vocabulary and/orset of grammar rules can be determined for the user that is appropriatefor the proficiency level of the user. For example, based on the pastinteractions of the user, as described herein, a secondary languageproficiency measure can be determined along with a set of terms,phrases, and/or grammar rules that the user has utilized in the pastinteractions. For example, a user may submit the query “How do you sayhello in French” multiple times, which may indicate that the user haslow proficiency in French and requires additional assistance intranslation and/or pronunciation of “hello” in French. Thus, based onmonitoring the past interactions of the user, a vocabulary of knownterms can be determined as well as a vocabulary of terms that the userhas not mastered but has indicated an interest in being providedmultiple times.

In some implementations, subsequent interactions of the user (i.e.,interactions that occur after the secondary language proficiency measurehas been initially determined) may be utilized to adjust the secondarylanguage proficiency measure of the user to reflect increased (ordecreased) proficiency of the user. For example, as a user becomes moreproficient in the secondary language, the user may submit queries and/orrequest translations that are more complex and/or include terms and/orgrammar that indicates a level of proficiency that is more indicative ofproficiency than previously determined. In those instances, thesecondary language proficiency of the user can be reevaluatedperiodically and/or when the user utilizes vocabulary that is notincluded in the determined vocabulary for a user with the initiallydetermined proficiency level.

In some implementations, the past interactions of the user that arerelated to the secondary language may be utilized to determine a userinterest measure for the user that is indicative of user interest inlearning the secondary language. For example, a user may submit one ormore phrases and request the one or more phrases be translated into thesecondary language. Also, for example, a user may interact with one ormore applications that are configured to assist in language learning.Also, for example, the user may browse webpages with content in thesecondary language, webpages related to learning the secondary language,and/or other resources that indicate the user has interest in a languageother than the primary language of the user.

By determining a secondary language proficiency measure for the user(and, optionally or alternatively, a user interest measure indicating aninterest by the user to be provided with additional resources directedto learning the secondary language), a secondary language learningapplication may be provided to the user. The secondary language learningapplication can be pre-configured and/or initialized based on thedetermined secondary language proficiency measure for the user such thatthe user is not required to complete a setup process to utilize thelanguage learning application. Thus, the user may be provided with anindication that the language learning application is available and beginusing the application without further processing and/or evaluation ofthe user language proficiency.

In some implementations, once a user language proficiency measure hasbeen determined, one or more parameters of the language learningapplication can be set with values based on the determined languageproficiency measure. Examples of parameters that may be set include, forexample, target vocabulary for the user proficiency, exercises that aredirected to grammatical rules that are appropriate for the userproficiency, and/or other aspects that are related to language learningthat may be inferred from the secondary language proficiency measure.When the user accesses the language learning application, tools that areprovided to the user can be tailored to the user's proficiency withoutrequiring the user to manually set one or more of the parameters.

In some implementations, a user may be provided with a graphical userinterface (GUI) with one or more parameters pre-set and/or one or morefields pre-filled with information that can be determined based on thesecondary language proficiency of the user. For example, a GUI may havea dropdown menu of potential languages and, based on the pastinteractions of the user, the determined user interest of the user inlearning a secondary language, and/or the user proficiency in asecondary language, the dropdown menu may be pre-filled with theidentified secondary language of interest. Also, for example, a slidingbar may be provided in the GUI that indicates, on a scale of values, theproficiency of the user, with the sliding bar set at a value that isappropriate for the determined secondary language proficiency measure ofthe user.

In some implementations, the secondary language proficiency measure fora user may be utilized to bias a speech recognition model to improverecognition of one or more terms and/or grammar rules for a secondarylanguage that a user may provide via an utterance. For example, a usermay utilize an automated assistant that in turn utilizes a speechrecognition model to translate audio data into a textual representationof the speech. In instances where a user utters one or more terms in asecondary language, the speech recognition model may be biased torecognize terms that are likely to be uttered by the user given thelanguage proficiency measure of the user. For example, a user may have asecondary language proficiency measure that indicates that the user hascomprehension of some basic vocabulary and/or phrases. The speechrecognition model that is utilized to process speech of the user may bebiased to additionally recognize terms in the secondary language thatwould not otherwise be given as much weight in typical use of the speechrecognition model. For example, the user may utter a simple Spanishphrase (e.g., “Que tiempo hace,” or “What time is it”) and, based on thesecondary language proficiency of the user, the speech recognition modelmay be biased to recognize the utterance of “que” as being a Spanishword that is likely to be used by the user in utterances. Thus, when theuser utters “Que tiempo hace,” the speech recognition model is morelikely to recognize “que” as the intended Spanish word and less likelyto identify the term as an English word, such as the letter “K” (a termwith the same pronunciation as “que”) that would otherwise be morelikely to be spoken by a user with a primary language preference set toEnglish.

In some implementations, the user may request translation of a phraseand be provided with the phrase translated into the secondary languageof interest to the user. For example, the user may invoke an automatedassistant and utter “how do you say ‘what time is it’ in Spanish.” Insome instances, the user may be provided with a textual indication ofthe translation along with an interactive icon that allows the user tohear the translation being spoken by a person in the secondary language(e.g., an audio clip of a user speaking in Spanish). However, ininstances where a user requests a longer phrase to be translated, theuser may have difficulty in practicing and/or understanding the entiretranslation, based on the current proficiency of the user in thesecondary language. In those instances, the translated phrase can besplit into smaller text blocks based on the secondary languageproficiency measure of the user. For example, if the user has asecondary language proficiency measure indicative of low proficiency,the user may be provided with multiple smaller texts, each of whichincludes only a single word or small phrase (e.g., a first subtext of“Que,” a second subtext of “tiempo,” and a third subtext of “hace). Foreach of the provided subtexts, the user may be provided with an icon tolisten to audio of the corresponding subtext. Similarly, a user with asecondary language proficiency measure that is indicative of a higherproficiency in the language, the translation (i.e., “que tiempo hace”)can be provided as a single text with translation icon based ondetermining that the user is more likely to comprehend longer phrases.

Implementations described herein allow for one or more tools to beautomatically tailored to the proficiency of the user while mitigatingadditional processing time that would otherwise be required to performsetup and initialization of language learning tools. For example, byprefilling one or more parameters of a GUI for a secondary languagelearning application, initial evaluation of user proficiency is notnecessary. Instead, the language proficiency of the user may bepassively determined based on the behavior of the user in the normalcourse of interactions of the user with one or more applications.Further implementations described herein allow for a speech recognitionmodel to be biased towards only those terms and/or grammar that theuser, given a particular proficiency measure, will likely utter ininteractions with, for example, an automated assistant. In thoseinstances, computing resources are not being expended to bias the speechrecognition model to recognize terms and/or grammar that the user isunlikely to include in queries, given the secondary language proficiencyof the user. Still further, implementations described herein reduce thelikelihood that a user will make repeated translation requests byproviding the translation in smaller subtexts. By splitting atranslation into smaller texts, the user is less likely to request thata given translation be repeated, thus reducing resources required toperform the translation and/or repeat the translation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an example environment in whichimplementations disclosed herein may be implemented.

FIG. 2 is an illustration of an example graphical user interface thatmay be utilized for one or more implementations described herein.

FIG. 3A and FIG. 3B are an illustration of an example graphical userinterface that may be utilized for one or more implementations describedherein.

FIG. 4 is a flowchart of one or more implementations described herein.

FIG. 5 is a flowchart of one or more additional implementationsdescribed herein.

FIG. 6 is a flowchart of one or more additional implementationsdescribed herein.

FIG. 7 illustrates an example architecture of a computing device.

DETAILED DESCRIPTION

Referring to FIG. 1 , an example environment is provided that may beutilized to implement one or more embodiments disclosed herein. Theexample environment includes a user device 100 that is executing alanguage learning application 105. Further, the user device 100 includesa microphone 101, a speaker 102, and a graphical user interface (GUI)103. Although illustrated in FIG. 1 as executing on user device 100, oneor more components of language learning application 105 may be executingremotely from the user device 100. For example, one or more componentsof language learning application 105 may have a local component that isexecuting on user device 100 and one or more remote components that areexecuting on another device that is in communication with the userdevice 100, such as on a server that is in network communication withthe user device 100.

The language learning application 105 includes a past interactionmanager 110 that can monitor interactions of the user with one or moreapplications. Further, the language learning application 105 includes alanguage proficiency manager 115 that can determine, based on pastinteractions of the user, a secondary language proficiency measureindicative of user proficiency with the secondary language. Stillfurther, the language learning application includes a user accountmanager 120 that includes account information for one or more users,such as parameters set by the user to indicate personal preferences.Further, language learning application 105 includes a language interestmanager that can determine user interest in learning a secondarylanguage. Further, the language learning application 105 includes agraphical user interface manager 130 to generate content and render thecontent via GUI 103 of the user device. User device 100 further includesan automatic speech recognizer 135, which can be utilized to receivespeech, such as an utterance of the user via microphone 101 and convertthe speech into one or more words for further processing. Speaker 102may be utilized by one or more components, such as a text-to-speechcomponent (not shown), to provide audio data to the user.

In some implementations, user account manager 120 can determine whethera particular language is the primary language of the user, or whether aparticular language is secondary to the primary language of the user.For example, in setting up the language learning application 105 and/orone or more accounts that are executing one the user device 100, theuser may be prompted to provide a primary language for future content.The account may be associated with the user device 100 (e.g., an initialaccount registration when the user device 100 is first accessed by theuser), with the language learning application 105 (e.g., an initialsetup of the application 105 when it is initially downloaded oraccessed), and/or one or more other accounts that are associated withthe user and can indicate preferences of the user.

In some implementations, a user may interact with one or moreapplications other than the language learning application. Pastinteraction manager 110 can analyze the interactions of the user todetermine whether the user has accessed one or more applications and/orrequested content that is related to a secondary language. For example,one or more databases may include stored information indicating useraccess of documents, query logs of interactions of the user with anautomated assistant, webpages that the user has browsed, indications ofmedia content that the has viewed, and/or other interactions of the userwith one or more applications that are external to the language learningapplication 105.

In some implementations, past interaction manager 110 may identify, fromthe stored past interactions, one or more interactions that are relatedto a secondary language. For example, a user may view a webpage and pastinteraction manager 110 can determine whether the webpage that wasviewed by the user is related to learning a secondary language and/orincludes content that is presented in a secondary language. Also, forexample, past interaction manager 110 may identify user interactionswith one or more applications other than the language learningapplication 105. For example, user device 100 may have a Spanishlanguage game application installed, and interactions of the user withthe game application (e.g., indications of the user downloading the gameapplication, accessing the game application) may be stored in one ormore databases and/or otherwise identified by the past interactionsmanager 110.

In some implementations, past interactions may include the user issuingone or more queries in a secondary language. For example, a user mayissue a textual query via a web browser (not shown) executing on theuser device 100. Past interaction manager 110 may identify the textualquery and determine that the query is in a secondary language (e.g., alanguage other than the primary language indicated by a user account ofthe user). Also, for example, a user may utter a query that is capturedby microphone 101 and processed by automatic speech recognizer 135. Pastinteraction manager 110 can determine, based on the processed audiodata, that at least a portion of the spoken utterance is in a secondarylanguage.

In some implementations, the user may submit a query that is in theprimary language of the user but related to a secondary language. Forexample, a user may utter the phrase “How do you say hello in Spanish,”indicating that the user has interest in being provided a translation ofa term from English to Spanish. Past interaction manager 110 candetermine that the query, although in English, is related to Spanish.Based on determining that the query is related to Spanish, pastinteraction manager 110 can store the interaction in one or moredatabases for further processing by one or more other components. Also,for example, a user may submit a query requesting to have a phrase orterm pronounced. For example, a user may submit a text query of “How doyou say hola” to an application (e.g., a web browser) and the user maybe provided with audio output via speaker 102 that includes audio of aspeaker (either a human speaker or a speech simulation) pronouncing theterm or phrase. The past interaction manager 110 may determine that thequery, which includes at least a portion in a primary language of theuser and a portion in a secondary language of the user, is related to asecondary language of the user.

In some implementations, past interaction manager 110 can identifyinteractions of the user with one or more applications. For example,user device 100 may have one or more language learning applications thatare executing on the device, and past interaction manager 110 canidentify that a user has accessed one or more of the applications. Also,for example, a user may utilize a web browser application executing onuser device 100 to browse to one or more webpages that include languagelearning information (e.g., web-based interactive games, study tools)and past interaction manager 110 can determine that the user hasaccessed information that is related to learning a secondary language.

In some implementations, past interaction manager 110 may determine thatthe user has interacted with one or more applications that are relatedto a secondary language and store information related to thoseinteractions in a database. In some implementations, past interactionmanager 110 may identify terms and/or grammar rules that the user hasutilized in interactions. For example, a user may submit a query of “quetiempo hace” and past interaction manager 110 may store indications ofthe vocabulary used by the user in a database. In some implementations,past interaction manager 110 may identify mistakes made by the user inutilizing a secondary language in interactions with one or moreapplications. For example, a user may submit a query of “time is it”instead of a well-formed query of “what time is it,” and pastinteraction manager 110 can store one or more indications of thevocabulary and/or grammar rules that the user utilized incorrectly. Insome implementations, a user may submit a request to translate a phraseinto a secondary language and past interaction manager 110 can store anindication of the terms and/or phrases that were provided to the user inresponse to the request to translate. In some implementations, pastinteraction manager 110 can identify terms and/or phrases that wereprovided to a user while interacting with other applications, such asterms that are included in webpages that are accessed by the user, termsthat were rendered to a GUI while the user is accessing a languagelearning application, and/or other vocabulary and/or concepts that theuser was exposed to while interacting with other applications.

Language proficiency manager 115 can determine, based on theinteractions identified by the past interaction manager 110, a secondarylanguage proficiency measure for the user. Language proficiency manager115 can determine, based on the interactions, terms and/or phrasesutilized during interactions, and/or responses to queries that arerelated to the secondary language to determine a secondary languageproficiency measure that is indicative of the user proficiency in thesecondary language. For example, past interaction manager 110 mayidentify an interaction of the user that includes the user submitting aquery with the term “hola” and language proficiency manager 115 candetermine a secondary language proficiency measure that indicates thatthe user has at least a minimal proficiency in Spanish. As the userinteracts with additional applications and past interaction manager 110identifies the interactions that are related to Spanish, the languageproficiency manager 115 can refine the secondary language proficiencymeasure for the user based on the additional terms and/or concepts thatthe user has utilized in queries, been provided by applications, and/orrequested in queries.

In some implementations, a user may request that a phrase be translated.The terms that are provided to the user in response to the request canbe utilized by language proficiency manager 115 to determine a secondarylanguage proficiency measure for the user. For example, if the userissues a query of “How do I say hello in Spanish,” language proficiencymanager 115 may determine a language proficiency measure that isindicative of a low proficiency in a language based on determining thata user with a high proficiency in Spanish would be unlikely to requiretranslation of the term “hello.” Also, for example, if a user issues arequest for a more complex phrase to be translated, language proficiencymanager 115 may determine a secondary language proficiency measure thatis more indicative of proficiency based on determining that a user withsome proficiency in a language may still likely require some translationassistance with more complex phrases. The terms of the resultingtranslation may further be utilized to determine a secondary languageproficiency measure for the user. For example, as the user receivestranslations, language proficiency manager 115 can utilize the resultingvocabulary to determine a secondary language proficiency measure thatreflects the vocabulary and/or phrases to which the user has beenexposed.

In some implementations, language proficiency manager 115 may determinea secondary language proficiency measure based on a quantity oftranslation requests submitted by the user. For example, if a userrequests translations for the same or similar terms multiple times,language proficiency manager 115 may determine that the user is learningthose terms and/or phrases but has not yet mastered their usage. Thus, asecondary language proficiency measure may be determined that indicatesthat those terms (or terms of similar complexity) are currently beinglearned by the user. As translation requests become more complex and/orrepeated translation requests become less frequent, language proficiencymanager 115 can update the secondary language proficiency measure forthe user to reflect the user improving proficiency in the secondarylanguage.

In some implementations, language proficiency manager 115 can determinea secondary language proficiency measure for a user based on the grammarand/or vocabulary that the user uses when sending a query in thesecondary language. For example, assuming the secondary language isEnglish, past interactions manager 110 may identify a first usersubmitting a query of “time it is” and language proficiency manager 115can determine a secondary language proficiency measure that isindicative of low proficiency based on the limited vocabulary and/orgrammatical errors of the submitted query. Also, for example pastinteraction manager 110 may identify a second user submitting a query of“what time is it” and language proficiency manager 115 may determine asecondary language proficiency measure for the second user that is moreindicative of proficiency than the first user based on determining thatthe second user has submitted a query that is grammatically correct.

In some implementations, language proficiency manager 115 can determinea secondary language proficiency measure based on interactions of theuser with one or more documents that include content in the secondarylanguage. For example, a user may access a news webpage that is in thesecondary language and browse the webpage. Language proficiency manager115 can determine what vocabulary the user has viewed, how long the userspent viewing the webpage and/or portions of the webpage, what termsfrom the webpage were submitted for translation, and/or otherindications of user exposure to the secondary language that may beindicative of proficiency of the user in the secondary language. Also,for example, a user may access a video that includes subtitles in thesecondary language (or, conversely, a video in a secondary language thathas subtitles in the primary language of the user). Language proficiencymanager 115 may determine a secondary language proficiency measure forthe user based on the length of time that the user watched the video,the vocabulary used in the video, whether the user activated thesubtitles, how long the user activated the subtitles, differences insubtitle usage between viewings of the video, and/or other interactionsthat can indicate a proficiency level of the user.

In some implementations, language proficiency manager 115 may identifyother errors that the user has made in submitting requests. For example,past interactions manager 110 may monitor the usage of autocorrect thatwas utilized by the user when submitting a query and/or typing a phrasein a document (e.g., word processor document, email). Languageproficiency manager 115 can determine a secondary language proficiencymeasure based on determining which terms and/or grammatical rules wereinitially incorrectly utilized, which autocorrection suggestions wereselected by the user, and/or the complexity of the suggestions that wereselected (e.g., the user selecting a correct and more complex suggestionthat the initially submitted terms and/or phrase).

In some implementations, a number of past interactions with one or moreapplications, other than language learning application 105, can beutilized by language proficiency manager 125 to determine a secondarylanguage proficiency measure for the user. For example, languageproficiency manager 115 may determine that the number of instances thata user interacted with an application, as described herein, satisfies athreshold. Thus, language proficiency manager 115 may determine that auser that has not interacted with applications in a manner related tothe secondary language a threshold number of times is less proficientthat a second user that has interacted with applications in a mannerrelated to the secondary language. As another example, languageproficiency manager 115 may determine a secondary language proficiencymeasure based on a percentage of interactions of the user that arerelated to the secondary language. As the percentage of interactionsincreases (i.e., the user provides and/or receives additionalinformation in the secondary language), language proficiency manager 115can determine a secondary language proficiency measure that is moreindicative of proficiency of the user for the secondary language.

In some implementations, past interaction manager 110 can identify anaudio query from the user that includes one or more terms in thesecondary language, and language proficiency manager 115 can determine asecondary language proficiency measure based on user pronunciation ofthe one or more terms. For example, language proficiency manager 115 maycompare the audio of the user uttering a phrase in the secondarylanguage to audio of a second user that is more proficient in thesecondary language uttering the same phrase. Based on the comparison,language proficiency manager 115 can determine the secondary languageproficiency measure that is more indicative of proficiency when the twopronunciations are similar, and a secondary language proficiency measurethat is less indicative of proficiency of the user when thepronunciation of the user is less similar to the pronunciation of a userthat is more proficient in the secondary language.

Language interest manager 125 can determine a user interest measureindicative of user interest learning a secondary language. Based on thedetermined user interest measure satisfying a threshold, the efficiencyof one or more subsequent user interactions can be improved, such asproviding the user with an interface to a secondary language learningapplication, biasing one or more ASR models, and/or determining whetherto provide the user with one or more translated phrases that have beensplit into sub-phrases based on the proficiency of the user. One or moresignals described herein for determining a secondary languageproficiency measure for the user may additionally and/or alternative beutilized to determine whether to improve efficiency of one or more userinteractions. Thus, one or more techniques described herein forimproving efficiency of user interactions that are related to asecondary language can be performed only in instances wherein the pastinteractions of the user indicate a likelihood that the user hasinterest in those interactions be improved. For example, an ASR modelmay be biased based on the secondary language proficiency measuredescribed herein, but may only be performed in instances where pastinteractions of the user indicate that the user experience would beimproved by biasing the speech model.

In some implementations, a secondary language interest measure can bedetermined based on queries that the user has submitted in a secondarylanguage. Initially, the user interest measure can be based on the usersubmitting queries. As the user submits additional queries, the qualityof the grammar in the queries and/or vocabulary utilized by the user insubmitting queries can be an indication of continued interest of theuser in learning a secondary language, which can be utilized todetermine to provide additional secondary language learning tools to theuser, bias an ASR model, and/or perform other actions that improve theefficiency of user interactions related to the secondary language. Insome implementations, additional queries that are related to inquiringpronunciations of terms and/or requesting translations of phrases and/orterms into the secondary language can be utilized to determine alanguage interest measure for the user.

In some implementations, user interactions with other applicationsand/or webpages (or requests for additional information related tolearning a secondary language) can be utilized to determine a userinterest in learning a secondary language. For example, a user mayaccess an application that is directed to learning a secondary languageand/or may browse one or more webpages that are directed to languagelearning. A secondary language interest measure can be determined basedon frequency of utilizing and/or viewing applications and/or webpages(e.g., increased usage can be an indication of continued interest).Further, decreased usage of secondary language learning applicationsand/or other resources can be an indication that the user no longer hasinterest in learning a secondary language, and the user interest measuremay be adjusted accordingly.

Once past interactions manager 110 identifies interactions of the userthat are related to learning a secondary language, and provides theinteractions to language proficiency manager 115 to determine asecondary language proficiency measure and/or to language interestmanager 125 to determine a user interest measure indicative of userinterest in learning a secondary language, efficiency of one or moresubsequent interactions can improve by adjusting and/or performing oneor more actions that are related to the user learning the secondarylanguage. For example, a language learning application may be populatedwith information that is indicative of user proficiency, and theapplication can be provided to the user (or an account registrationinterface for the application) based on the user interest measuresatisfying a threshold interest level. Also, for example, an ASR modelcan be adjusted to bias to one or more terms and/or vocabulary rulesthat are indicative of user proficiency in the secondary language, suchas biasing the ASR to recognize terms that are commonly utilized by auser with the determined secondary language proficiency measure. Also,for example, translations of terms and/or phrases initiated by the usermay be provided to the user in sub-phrases that are split based on thesecondary language proficiency measure of the user.

In some implementations, one or more parameters of a plurality ofadjustable parameters of an interface for a language learningapplication can be pre-filled with values based on the determinedsecondary language proficiency measure that has been determined for theuser. For example, referring to FIG. 2 , an interface 200 of a languagelearning application is provided. Once language proficiency manager 115determines a secondary language proficiency measure (and, in someinstances, language interest manager 125 has determined user interest inbeing provided a secondary language learning application), the interface200 can be provided to the user. As illustrated, interface 200 includesa language selection section 205 that has been pre-filled with the“Spanish” radio button selected. This selection can be determined basedon, for example, past interaction manager 110 identifying pastinteractions of the user that are related to Spanish language learning.Further, the interface 200 includes a user language level slide bar 210that is set to a value between beginner and intermediate. This can bedetermined by a component that shares one or more characteristics withlanguage proficiency manager 115, as described herein.

Interest category selection area 205 of interface 200 includes aplurality of types of content that a user may be interested in beingprovided. As illustrated, categories of “translations” and “websites”have been selected. The auto-filled selections of the content can bebased on, for example, determining what the user has viewed and/orinteracted with in past interactions. For example, a user may issue athreshold number of translation requests in past interactions, and“translations” may be automatically selected in the interest categoryselection area for the user based the threshold quantity of translationrequests. In some implementations, interest category selections may notbe provided to the user and instead the user may be automaticallyprovided with a variety of content, such as quizzes via the secondarylanguage learning application, secondary language websites, videos,and/or other content.

Button 215 allows the user to indicate confirmatory input to submit theparameters of the interface 200 in order to initialize and/or update alanguage learning application. In some implementations, the user may beprovided with the interface automatically, given an opportunity toadjust one or more of the parameters, and submit the values of theparameters to then be provided with the language learning application.The provided language learning application may then provide content tothe user based on the secondary language proficiency of the user and/orthe interests of the user. The language learning application mayprovide, among other content, games, puzzles, quizzes, videos,pronunciations of vocabulary, and/or other content that can assist theuser in learning the secondary language.

In some implementations, the user may be provided with prompts and/orother indications of the language learning application based on the userinterest measure of the user. For example, based on identifying that theuser is interacting with the language learning application and/orcontinuing to interact with other sources of secondary languageinformation, the user may be provided with a reminder (e.g., dailynotification on a smart device) to utilize the language learningapplication. Also, for example, if the user interest measure of the userchanges to indicate a decreased interest in being provided withadditional information related to learning a secondary language,notifications and/or other indications may be provided less frequentlyand/or less prominently. Conversely, if the user interacts withsecondary language learning sources less frequently, the user may beprovided with more frequent notifications to remind the user of thelanguage learning application and encourage continued study of thesecondary language.

In some implementations, the user may be provided with the interface 200and/or a similar interface when one or more measures for the user areadjusted. For example, the user may become more proficient in thesecondary language as the user interacts with the language learningapplication and/or other sources. As past interactions manager 110identifies the continued interactions of the user, language proficiencymanager 115 may determine that the language proficiency of the user hasincreased (e.g., use of additional vocabulary, viewing secondarylanguage videos without translations, better grammar in queries).Language proficiency manager 115 may provide updated values for one ormore parameters to graphical user interface manager 130, which mayprovide the user with interface 200 (or a similar interface) to allowthe user to view and/or adjust the updated values for the parameters.For example, graphical user interface manager 130 may provide interface200 with the slide bar 210 indicating an “intermediate” proficiencylevel. The user may select button 215 to continue with the adjustedvalues and/or adjust the values manually before continuing use of thelanguage learning application.

Automatic speech recognizer 135 receives audio data that includes a useruttering a phrase, such as via microphone 101, and recognizes speech forfurther processing. For example, a user can utter a phrase that iscaptured by microphone 101, automatic speech recognizer 135 can processthe speech to determine the intent and/or terms in the phrase, and oneor more components can further process the recognized phrase. This caninclude, for example, interacting with one or more applications,utilizing speech to text (STT) to determine a textual representation ofthe speech, and/or performing translations of terms into a secondarylanguage, as described herein.

In some implementations, language proficiency manager 125 can determinea secondary language proficiency measure that can be utilized toincrease the accuracy of automatic speech recognition (ASR) that isperformed by automatic speech recognizer 135 on secondary languagespoken utterances of the user that are in the secondary language. Forinstance, based on the secondary language proficiency measure, automaticspeech recognizer 135 (or another component that can provide automaticspeech recognizer with one or more terms to bias in speech recognition)select, from a superset of terms for the secondary language, a subset ofterms for the secondary language. Automatic speech recognizer 135 canutilize the selected subset of terms to bias an automatic speechrecognition model that is configured to recognize speech that isperformed in the secondary language utterances of the user. The subsetof terms can include, for example, terms that the user is likely to belearning based on the secondary language proficiency measure and canoptionally exclude term(s) that the user is unlikely to be learningand/or has already mastered. The secondary language ASR can be utilizedin processing spoken utterances of the user that are provided ininteractions with the language learning application, in generalinteractions of the user with an automated assistant, and/otherinteractions of the user.

In some implementations, the secondary language proficiency measure thatis determined by language proficiency manager 115 can be utilized toprovide a user with translations of terms and/or phrases of interestthat are provided by the user. The user can provide a query, eithertextually and/or audibly, that may require a response that is in thesecondary language. For example, the user may submit the query “How doyou say good morning in Spanish.” The request to translate one or moreterms into the secondary language can result in phrases of varyinglengths, based on the complexity of the request. In some instances, atranslated phrase may be better comprehended by the user if thetranslation (or other responses that are in the secondary language) aresplit into smaller portions and provided to the user separately. Phrasesmay be split based on the secondary proficiency measure of the user suchthat more proficient speakers can be provided with larger portions ofthe response than a less proficient speaker. For example, naturallanguage output text can be split into a plurality of subtexts, where aquantity of the subtexts and/or an amount of text in each of thesubtexts can be determined based on the secondary language proficiencymeasure. Generally, more subtexts and/or smaller size subtexts will beprovided with a low secondary language proficiency measure while lessersubtexts and/or larger size subtexts will be provided with a highersecondary language proficiency measure. This can make the output moreunderstandable (through more and/or smaller subtexts) for a lesserproficiency user, while making the rendering shorter (in the case ofaudible rendering) and/or take up less screen real estate (in the caseof visual rendering) for a higher proficiency user.

Referring to FIGS. 3A and 3B, example graphical user interfaces areillustrated that may be utilized with one or more implementationsdescribed herein. Interface 300 may be provided to a user that has alanguage proficiency score that is more indicative of languageproficiency than interface 350. In some implementations, one or morecomponents may be absent from an interface that is provided to a userand/or provided interface(s) may have additional components notillustrated in FIGS. 3A and 3B

Query box 305 includes a query that is provided to a user. In someimplementations, the query may be included in audio data that iscaptured via microphone 101 and processed by automatic speech recognizer135 and/or a text-to-speech model. In some implementations, the querymay be provided by the user as text via one or more interfaces and/orvia interface 300. Similarly, query box 355 includes a query that may beprovided by a user via speech and/or via a text query.

In some implementations, a query may include an indication that the userhas interest in translating a term and/or phrase into a secondarylanguage. As illustrated, the query “Good day. How are you? What time isit?” has been provided by the user into an interface that is configuredto receive terms and/or phrases for translation. In someimplementations, the user may provide a query via a general interface(e.g., graphical user interface, automated assistant) and indicate inthe query the interest in having the term and/or phrase translated. Forexample, a user may utter the query “translate good day into Spanish”and, in response, can be provided with interface 350.

Referring to interface 300, the user has requested a translation of“good day. How are you? What time is it?” and has been provided with atranslation of the phrase as three subtexts 310, 315, and 320. In someimplementations, the language proficiency measure for the user may beutilized to determine how to split the translation (or other response inthe secondary language) into the subtexts. As illustrated, the user isprovided with a first subtext 310 “Buenos dias” as a single phrase withmultiple terms. In some implementations, the first subtext 310 may bedetermined based on terms and/or phrases that the user has previouslyencountered in past interactions, terms and/or phrases of a reasonablelength for a typical user with the proficiency of the user, and/or otherindications that may be inferred from the language proficiency measure.In some implementations, the first subtext may be determined based onhow proficient the user is in the secondary language, with higherproficiency resulting in larger and/or more complex subtexts. The secondsubtext 315 includes the translation “Como te llamas” and the thirdsubtext 320 includes the translation “Que hora es.” Thus, for thelanguage proficiency measure of the user, each of the sentences of thequery has been provided as a separate subtext.

In some implementations, the user may be provided with more or fewersubtexts based on the user language proficiency measure. For example,referring to interface 350, the user has submitted the phrase “goodday,” which has been split into two subtexts 360 and 365. As an example,the user that submitted the phrase in interface 350 may have a lowerproficiency measure such that, instead of being provided with “buenosdias” as a single subtext, such as subtext 310, the user has beenprovided with each word separately. Thus, because the user thatsubmitted query 355 has a lower language proficiency measure, subtextsmay be shorter, may have new and/or complex vocabulary provided as itsown subtext, and/or may otherwise be split and provided to the userbased on the determined language proficiency measure for the user.

In addition to a textual translation of the query 305, each of thesubtexts is provided with an icon 325 that, when interacted with, canprovide the user with audio, via speaker 102, of the subtexts beinguttered. For example, if a user interacts with icon 325 (e.g., touchesthe icon on a touchscreen), the user can be provided with audio of aspeaker uttering “buenos dias.” Similarly, if a user interacts with icon370, the user may be provided with audio that includes a speakeruttering “buenos.” Audio may be generated from, for example, a recordingof a user uttering the phrase and/or via a text-to-speech component.

Referring to FIG. 4 , a flowchart 400 is provided that illustratesimplementations described herein. In some implementations, one or moresteps may be absent, one or more additional steps may be performed,and/or the illustrated steps may be performed in a different order. Forexample, as illustrated, step 410 is an optional step that may beperformed before step 405 and/or may not be performed at all.

At step 405, a language proficiency measure is determined based on pastinteractions of the user. Past interactions may be identified by acomponent that shares one or more characteristics with past interactionsmanager 110 of FIG. 1 . Further, a secondary language proficiencymeasure can be determined by language proficiency manager 115, asdescribed herein. In some implementations, past interactions of the usermay include resources that are viewed by the user, such as webpages andvideos, queries submitted by the user, such as search requests andrequests for translations, and/or interactions with other applications.

In some implementations, the language proficiency measure can bedetermined, based at least in part, on similarity between one or more ofthe past interactions. For example, a user may issue the sametranslation query repeatedly, which may indicate that the terms of thetranslation are currently being learned by the user. Also, for example,repeated viewing of a video in a secondary language of the user may beindicative of the proficiency of the user in the secondary language.Also, for example, a user viewing multiple webpages that are of asimilar proficiency level (e.g., viewing multiple webpages with beginnervocabulary) may be indicative of the proficiency level of the user.

In some implementations, the language proficiency measure can bedetermined, based at least in part, on complexity of one or more of thepast interactions. For example, a low proficiency measure may bedetermined, at least in part, for a user that issues beginner querieswith low complexity (e.g., “How do you say hello in Spanish”). Also, forexample, a higher proficiency measure may be determined, at least inpart, for a user that issues more complex queries (e.g., “How do you say‘what will the weather be like next Thursday’ in Spanish”).

In some implementations, the language proficiency measure can bedetermined, at least in part, by identifying errors in one or more pastinteractions of the user. For example, a user may submit a query in thesecondary language that includes one or more grammatical errors,incorrect usage of terms, and/or other errors that are less likely to bein queries submitted by user with a primary language of the secondarylanguage of the user. As errors in queries are less frequent and/or morecomplex errors, the secondary proficiency measure of the user may beupdated to a value that is more indicative of proficiency. Thus, a userthat makes fewer errors may have a higher proficiency measure than asecond user who makes more errors and/or makes more frequent errors.Further, the types of errors that are made by the user may be utilizedto determine what type of content to provide to the user and/or totarget particular content to assist the user in learning the secondarylanguage.

At optional step 410, a user interest measure is determined for the userbased on the past interactions of the user. The past interactions thatare utilized to determine the user interest measure can include one ormore of the past interactions that may be utilized to determine thelanguage proficiency measure of step 405. In some implementations, theuser interest measure can be determined by a component that shares oneor more characteristics with language interest manager 125. For example,as the quantity of interactions related to the secondary languageincreases, the user interest measure may increase to indicate continuedand/or increasing interest in being provided with a language learningapplication.

At step 415, values for one or more parameters for a graphical userinterface of a language learning application are automatically set. Forexample, a component that shares one or more characteristics withgraphical interface manager 130 may generate an interface that sharesone or more characteristics with interface 200 of FIG. 2 and describedherein. In some implementations, the parameters may include a userproficiency level that may be preset in the interface such that, whenthe language learning application is provided to the user, the contentthat is provided is appropriate for the learning level of the user. Atstep 420, confirmatory input is received from the user. This mayinclude, for example, the user interacting with the provided interface200, such as by selecting a button, such as button 215 of FIG. 2 .

At step 425, the values that were set in step 415 are assigned to anaccount of the user. The values may be assigned to the user by acomponent that shares one or more characteristics with user accountmanager 120. For example, once the user confirms the prefilled values ofinterface 200, the values can be store by user account manager 120 forusage by the language learning application.

Referring to FIG. 5 , a flowchart 500 is provided that illustratesimplementations described herein. In some implementations, one or moresteps may be absent, one or more additional steps may be performed,and/or the illustrated steps may be performed in a different order. Forexample, as illustrated, step 510 is an optional step that may beperformed before step 405 and/or may not be performed at all.

At step 505, a language proficiency measure is determined based on pastinteractions of the user. This step can share one or morecharacteristics with step 405 of FIG. 4 . For example, previousinteractions of the user may include identifying resources that wereviewed by the user, previous queries of the user, previous translationrequests of the user, and/or other interactions described herein.

At optional step 510, a user interest measure is determined for the userbased on the past interactions of the user. Step 510 may share one ormore characteristics with step 410 of FIG. 4 . For example, based ondetermining a user interest measure from past interactions of the user,one or more subsequent steps can be performed. Thus, by taking intoaccount a user interest measure, additional steps may be optionallyperformed only when the user interest satisfies a threshold.

At step 515, a subset of terms are determined for the user in theparticular language. The terms may include one or more terms that theuser has utilized in past interactions and/or terms that are appropriatefor the language proficiency measure of the user that was determined atstep 505. For example, the user may issue a query in the particularlanguage a threshold number of times and/or otherwise indicate commonusage of terms that can be included in the subset of terms in theparticular language.

At step 520, the particular grammar that includes the subset of terms isused to bias automatic speech recognition. Speech recognition can beperformed by a component that shares one or more characteristics withautomatic speech recognizer 135. For example, an automatics speechrecognition model may be configured to recognize terms of the particulargrammar such that, in instances where the user utters a phrase thatincludes the particular language (as opposed to the primary language ofwhich the ASR is configured to recognize), the ASR can more likelyrecognize the terms in the particular language.

Referring to FIG. 6 , a flowchart 600 is provided that illustratesimplementations described herein. In some implementations, one or moresteps may be absent, one or more additional steps may be performed,and/or the illustrated steps may be performed in a different order. Forexample, as illustrated, step 610 is an optional step that may beperformed before step 405 and/or may not be performed at all.

At step 605, a language proficiency measure is determined based on pastinteractions of the user. This step can share one or morecharacteristics with step 405 of FIG. 4 . For example, previousinteractions of the user may include identifying resources that wereviewed by the user, previous queries of the user, previous translationrequests of the user, and/or other interactions described herein.

At optional step 610, a user interest measure is determined for the userbased on the past interactions of the user. Step 610 may share one ormore characteristics with step 410 of FIG. 4 . For example, based ondetermining a user interest measure from past interactions of the user,one or more subsequent steps can be performed. Thus, by taking intoaccount a user interest measure, additional steps may be optionallyperformed only when the user interest satisfies a threshold.

At step 615, a text in a secondary language is identified. The text caninclude a request to translate a phrase into a secondary language and/orotherwise be a request that can result in a phrase in the secondarylanguage. As an example, the user may submit a request of “How do yousay good morning in Spanish.” In some implementations, the requestedtext can be translated into Spanish for further processing, as describedherein.

At step 620, a plurality of subtexts are determined for the text in thesecondary language based on the language proficiency measure of theuser. The length of one or more of the subtexts is determined based onthe language proficiency measure of the user such that a user with alower language proficiency measure may be provided subtexts that areshorter in length than a user with a high language proficiency measure.For example, for a text of “buenos dias,” a first user may be providedthe text as a single subtext whereas a second user with a lower languageproficiency measure may be provided with a first subtext of “buenos” anda second subtext of “dias.”

At step 625, each subtext is rendered as a separate block in a userinterface that is provided to the user. For example, subtexts may berendered by a component that shares one or more characteristics withgraphical user interface manager 130 and may provide an interface thatshares one or more characteristics with interface 300 and/or 350 ofFIGS. 3A and 3B. In some implementations, the subtexts may be providedwith an interactive icon that allows the user to listen to a speakeruttering the associated subtexts. For example, icons 325 and 370 ofFIGS. 3A and 3B may, when selected, cause the client device to provideaudio of the related subtexts 310 and 360.

Although FIGS. 4 to 6 describe three implementations that utilize asecondary language proficiency measure to improve efficiency ofsubsequent interactions of the user, additional implementations mayutilize any combination of the steps that are described in FIGS. 4 to 6. For example, in some implementations, a secondary language proficiencymeasure can be utilized to automatically set values for one or moreparameters of an interface of a language learning application andfurther be utilized to bias an automatic speech recognition (ASR) model,as described in FIGS. 4 and 5 . Also, for example, a secondary languageproficiency measure can be utilized to both bias an ASR model and toprovide subtexts of a text to a user, as described in FIGS. 5 and 6 .Also, for example, a secondary language proficiency measure can be usedto automatically set values to parameters of a language learningapplication interface as well as to provide subtexts to a text, asdescribed herein and illustrated in FIGS. 4 and 6 . Also, for example, asecondary language proficiency measure can be utilized to perform thesteps of FIGS. 4, 5 and 6 in combination.

FIG. 7 is a block diagram of an example computer system 710. Computersystem 710 typically includes at least one processor 714 whichcommunicates with a number of peripheral devices via bus subsystem 712.These peripheral devices may include a storage subsystem 724, including,for example, a memory 725 and a file storage subsystem 726, userinterface output devices 720, user interface input devices 722, and anetwork interface subsystem 716. The input and output devices allow userinteraction with computer system 710. Network interface subsystem 716provides an interface to outside networks and is coupled tocorresponding interface devices in other computer systems.

User interface input devices 722 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 710 or onto a communication network.

User interface output devices 720 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 710 to the user or to another machine or computersystem.

Storage subsystem 724 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 724 may include the logic toperform selected aspects of method 400, method 500, and/or to implementone or more of client device 100, query processing application 105, anoperating system executing query processing application 105 and/or oneor more of its components, an automated assistant, and/or any otherengine, module, chip, processor, application, etc., discussed herein,method 400, method 500, method 600, and/or to implement one or more ofuser device 100, language learning application 105 and/or one or more ofits components, and/or any other engine, module, chip, processor,application, etc., discussed herein.

These software modules are generally executed by processor 714 alone orin combination with other processors. Memory 725 used in the storagesubsystem 724 can include a number of memories including a main randomaccess memory (RAM) 730 for storage of instructions and data duringprogram execution and a read only memory (ROM) 732 in which fixedinstructions are stored. A file storage subsystem 726 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 726 in the storage subsystem 724, or inother machines accessible by the processor(s) 714.

Bus subsystem 712 provides a mechanism for letting the variouscomponents and subsystems of computer system 710 communicate with eachother as intended. Although bus subsystem 712 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computer system 710 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computer system 710depicted in FIG. 7 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputer system 710 are possible having more or fewer components thanthe computer system depicted in FIG. 7 .

In some implementations, a method implemented by one or more processorsis provided and includes determining, based on processing secondarylanguage interaction data of a user, a language proficiency measure thatis specific to the user and that is specific to a secondary languagethat is not indicated as a primary language for an account of the user.The secondary language interaction data of the user is based on pastinteractions of the user, via one or more client devices associated withthe account of the user, that are related to the secondary language, andthe past interactions are not directed to a language learningapplication. The method further includes automatically setting, based onthe language proficiency measure, a value for a parameter of a pluralityof adjustable parameters of the settings GUI and receiving confirmatoryuser interface input, of the user, that is provided at the settings GUI,wherein the confirmatory user interface input confirms values for theadjustable parameters, including confirming the automatically set valuefor the parameter. Finally, the method further includes assigning thevalues to the account of the user, wherein the values, when assigned tothe account of the user, dictate output rendered by the languagelearning application in subsequent interactions with the user.

These and other implementations of the technology disclosed herein caninclude one or more of the following features.

In some implementations, the method further includes determining, basedon the past interactions of the user, a user interest measure indicativeof user interest in learning the secondary language, determining thatthe user interest measure satisfies a threshold, and, in response todetermining the user interest measure satisfies a threshold, and priorto the user accessing the settings GUI, causing a notification to beprovided, at one of the client devices, wherein the notification, whenselected, causes the language learning application to be accessed, andwherein the user is accessing the settings GUI responsive to selectingthe notification. In some of those implementations, determining the userinterest measure comprises determining the user interest measure basedon a quantity of the past interactions that are related to the secondarylanguage. In some implementations, determine the user interest measureincludes determining the user interest measure based on identifying theuser accessing media content in the secondary language

In some implementations, determining the language proficiency measureincludes determining the language proficiency measure based on aquantity of the past interactions that are related to the secondarylanguage.

In some implementations, determining the language proficiency includesdetermining the language proficiency measure based on a quantity of thepast interactions that are related to the secondary language.

In some implementations, determining the language proficiency includesdetermining the language proficiency measure based on similarity betweenone or more of the past interactions.

In some implementations, determining the language proficiency measureincludes determining the language proficiency measure based oncomplexity of one or more of the past interactions.

In some implementations, determining the language proficiency measureincludes identifying errors in one or more past interactions of theuser.

In some implementations, the secondary language interaction dataincludes audio of the user uttering one or more terms in the secondarylanguage, determining the language proficiency measure includesdetermining the language proficiency measure based on accuracy of theuser in speaking the one or more terms in the secondary language.

In some implementations, the method further includes determining, basedon processing subsequent secondary language interaction data of a user,an updated language proficiency measure, and updating the valuesassigned to the account of the user.

In other implementations, a method implemented by one or more processorsis provided and includes determining, based on processing secondarylanguage interaction data of a user, a language proficiency measure thatis specific to the user and that is specific to a secondary languagethat is not specified as a primary language for an account of the user,wherein the secondary language interaction data of the user is based onpast interactions of the user, via one or more client devices associatedwith the account of the user, that are related to the secondarylanguage, and wherein the past interactions are not directed to alanguage learning application, selecting, from a superset of terms forthe secondary language and based on the language proficiency measure, asubset of the terms to include in the particular grammar for the user,and in response to selecting the particular grammar for the user, usingthe particular grammar in biasing automatic speech recognition of aspoken utterance of the user, wherein the automatic speech recognitionis performed using a speech recognition model for the secondarylanguage.

In some of those implementations, the method further includes receivinga spoken query from the user, wherein the spoken query includes one ormore terms of the grammar, determining, utilizing the one or more speechrecognition models, a textual representation of the spoken query, anddetermining a response to the spoken query. In others of thoseimplementations, the response includes a suggestion for pronunciation ofone or more of the terms of the grammar that are included in the spokenquery. In others of those implementations, the response includes asuggestion for correcting one or more grammatical errors in the spokenquery.

In some implementations, the method further includes receiving one ormore additional queries from the user, wherein the one or moreadditional queries include one or more terms in the secondary languagethat are not part of the selected grammar, adjusting the languageproficiency score based on the one or more terms in the secondarylanguage that are not part of the selected grammar, and adding one ormore of the terms in the secondary language to the grammar.

In yet others of the implementations, a method implemented by one ormore processors is provided and includes determining, based onprocessing secondary language interaction data of a user, a languageproficiency measure that is specific to the user and that is specific toa secondary language that is not specified as a primary language for anaccount of the user, wherein the secondary language interaction data ofthe user is based on past interactions of the user, via one or moreclient devices associated with the account of the user, that are relatedto the secondary language; identifying text in the secondary language;determining a plurality of subtexts for the text, wherein a size of atleast one of the subtexts is determined based on the secondary languageproficiency measure; and rendering each subtext as a separate block in auser interface along with a corresponding interactive icon that, wheninteracted with, causes the user to be provided with audio of thecorresponding text being read.

In some of those implementations, the method further includes receivingaudio of a user uttering a phrase, generating a response to the phrase,wherein the response is in the secondary language, and generating thetext based on the response. In some of those implementations, generatingthe response includes translating the phrase into the secondarylanguage, wherein the text is a translated version of the phrase.

Some implementations include one or more processors (e.g., centralprocessing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s), and/ortensor processing unit(s) (TPU(s)) of one or more computing devices,where the one or more processors are operable to execute instructionsstored in associated memory, and where the instructions are configuredto cause performance of any of the methods described herein. Someimplementations additionally or alternatively include one or morenon-transitory computer readable storage media storing computerinstructions executable by one or more processors to perform any of themethods described herein.

In situations in which the systems described herein collect personalinformation about users (or as often referred to herein,“participants”), or may make use of personal information, the users maybe provided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current geographic location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. Also, certain data may be treated in one or more waysbefore the data is stored or used, so that personal identifiableinformation is removed. For example, a user's identity may be treated sothat no personal identifiable information can be determined for theuser, or a user's geographic location may be generalized wheregeographic location information is obtained (such as to a city, ZIPcode, or state level), so that a particular geographic location of auser cannot be determined. Thus, the user may have control over howinformation is collected about the user and/or used.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

1. A method implemented by one or more processors, the methodcomprising: determining, based on processing secondary languageinteraction data of a user, a language proficiency measure that isspecific to the user and that is specific to a secondary language thatis not indicated as a primary language for an account of the user,wherein the secondary language interaction data of the user is based onpast interactions of the user, via one or more client devices associatedwith the account of the user, that are related to the secondarylanguage, and wherein the past interactions are not directed to alanguage learning application; responsive to the user accessing asettings graphical user interface (GUI) of the learning application:automatically setting, based on the language proficiency measure, avalue for a parameter of a plurality of adjustable parameters of thesettings GUI; receiving confirmatory user interface input, of the user,that is provided at the settings GUI, wherein the confirmatory userinterface input confirms values for the adjustable parameters, includingconfirming the automatically set value for the parameter; and inresponse to receiving the confirmatory user interface input: assigningthe values to the account of the user, wherein the values, when assignedto the account of the user, dictate output rendered by the languagelearning application in subsequent interactions with the user.
 2. Themethod of claim 1, further comprising: determining, based on the pastinteractions of the user, a user interest measure indicative of userinterest in learning the secondary language; determining that the userinterest measure satisfies a threshold; and in response to determiningthe user interest measure satisfies a threshold, and prior to the useraccessing the settings GUI: causing a notification to be provided, atone of the client devices, wherein the notification, when selected,causes the language learning application to be accessed, and wherein theuser is accessing the settings GUI responsive to selecting thenotification.
 3. The method of claim 2, wherein determining the userinterest measure comprises: determining the user interest measure basedon a quantity of the past interactions that are related to the secondarylanguage.
 4. The method of claim 2, wherein determining the userinterest measure comprises: determining the user interest measure basedon identifying the user accessing media content in the secondarylanguage.
 5. The method of claim 1, wherein determining the languageproficiency measure comprises: determining the language proficiencymeasure based on a quantity of the past interactions that are related tothe secondary language.
 6. The method of claim 1, wherein determiningthe language proficiency measure comprises: determining the languageproficiency measure based on similarity between one or more of the pastinteractions.
 7. The method of claim 1, wherein determining the languageproficiency measure comprises: determining the language proficiencymeasure based on complexity of one or more of the past interactions. 8.The method of claim 1, wherein determining the language proficiencymeasure comprises: identifying errors in one or more past interactionsof the user.
 9. The method of claim 1, wherein the secondary languageinteraction data includes audio of the user uttering one or more termsin the secondary language, and wherein determining the languageproficiency measure comprises: determining the language proficiencymeasure based on accuracy of the user in speaking the one or more termsin the secondary language.)
 10. The method of claim 1, furthercomprising: determining, based on processing subsequent secondarylanguage interaction data of a user, an updated language proficiencymeasure; and updating the values assigned to the account of the user.11. A method implemented by one or more processors, the methodcomprising: determining, based on processing secondary languageinteraction data of a user, a language proficiency measure that isspecific to the user and that is specific to a secondary language thatis not specified as a primary language for an account of the user,wherein the secondary language interaction data of the user is based onpast interactions of the user, via one or more client devices associatedwith the account of the user, that are related to the secondarylanguage, and wherein the past interactions are not directed to alanguage learning application; selecting, from a superset of terms forthe secondary language and based on the language proficiency measure, asubset of the terms to include in the particular grammar for the user;and in response to selecting the particular grammar for the user: usingthe particular grammar in biasing automatic speech recognition of aspoken utterance of the user, wherein the automatic speech recognitionis performed using a speech recognition model for the secondarylanguage.
 12. The method of claim 11, further comprising: receiving aspoken query from the user, wherein the spoken query includes one ormore terms of the grammar; determining, utilizing the one or more speechrecognition models, a textual representation of the spoken query; anddetermining a response to the spoken query.
 13. The method of claim 12,further comprising: providing the response to the user, wherein theresponse includes one or more terms of the grammar.
 14. The method ofclaim 12, wherein the response includes a suggestion for pronunciationof one or more of the terms of the grammar that are included in thespoken query.
 15. The method of claim 12, wherein the response includesa suggestion for correcting one or more grammatical errors in the spokenquery.
 16. The method of claim 11, further comprising: receiving one ormore additional queries from the user, wherein the one or moreadditional queries include one or more terms in the secondary languagethat are not part of the selected grammar; adjusting the languageproficiency score based on the one or more terms in the secondarylanguage that are not part of the selected grammar; and adding one ormore of the terms in the secondary language to the grammar.
 17. A methodimplemented by one or more processors, the method comprising:determining, based on processing secondary language interaction data ofa user, a language proficiency measure that is specific to the user andthat is specific to a secondary language that is not specified as aprimary language for an account of the user, wherein the secondarylanguage interaction data of the user is based on past interactions ofthe user, via one or more client devices associated with the account ofthe user, that are related to the secondary language; identifying textin the secondary language; determining a plurality of subtexts for thetext, wherein a size of at least one of the subtexts is determined basedon the secondary language proficiency measure; and rendering eachsubtext as a separate block in a user interface along with acorresponding interactive icon that, when interacted with, causes theuser to be provided with audio of the corresponding text being read. 18.The method of claim 17, further comprising: receiving audio of a useruttering a phrase; generating a response to the phrase, wherein theresponse is in the secondary language; and generating the text based onthe response.
 19. The method of claim 18, wherein generating theresponse comprises: translating the phrase into the secondary language,wherein the text is translated version of the phrase.